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CROSS-REFERENCE TO RELATED APPLICATION 

[0001] The present application is related to concurrently 

filed and commonly assigned U.S. patent application bearing 
serial no. xx/xxx,xxx (Attorney Docket No. M-11119 US), 
entitled "Method And Apparatus for Storing Image Information 
for Multiple Sampling Operations in a Digital Pixel Sensor/' 
by Benjamin P. Olding and Justin Reyneri. 

FIELD OF THE INVENTION 

[0002] The invention relates image sensor systems, and in 
particular, the present invention relates to a method for 
storing image information in a digital image sensor. 

BACKGROUND OF THE INVENTION 

[0003] A CMOS image sensor with pixel level analog-to- 
digital conversion is described in U.S. Patent No. 5,461,425 
of B. Fowler et al . (the M25 patent). Such an image 
sensor, referred to as a digital pixel sensor (DPS) , 
provides a digital output signal at each pixel element 
representing the light intensity detected by that pixel 
element. The combination of a photodetector and an analog- 
to-digital (A/D) converter in an area image sensor helps 
enhance detection accuracy and reduce power consumption, and 
improves overall system performance . 

[0004] In the DPS array of the M25 patent, the analog- 

to-digital conversion (ADC) is based on first order sigma 
delta modulation. While this ADC approach requires fairly 
simple and robust circuits, it has the disadvantages of 
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producing too much data and suffering from poor low light 
performance. U.S. Patent No. 5,801,657 of Fowler et al . , 
and U.S. Patent Application Serial No. 09/274,202 provide 
alternative ADC mechanisms that can significantly improve 
5 the overall system performance while minimizing the size of 
the A/D converters. The aforementioned patents and patent 
application are incorporated herein by reference in their 
entireties . 

[0005] Copending and commonly assigned U.S. patent 

10 application No. 09/567,638, entitled "Integrated Digital 
Pixel Sensor Having a Sensing Area and a Digital Memory 
Area" of David Yang et al . , describes an integrated DPS 
sensor with an on-chip memory for storing at least one frame 
of pixel data. The incorporation of an on-chip memory in a 
15 DPS sensor alleviates the data transmission bottleneck 

problem associated with the use of an off -chip memory for 
storage of the pixel data. In particular, the integration 
of a memory with a DPS sensor makes feasible the use of 
multiple sampling for improving the quality of the captured 
20 images. Multiple sampling is a technique capable of 
achieving a wide dynamic range without many of the 
disadvantages associated with other dynamic range 
enhancement techniques, such as degradation in signal-to- 
noise ratio and increased implementation complexity. 

2 5 Copending and commonly assigned U.S. patent application No. 

09/567,786, entitled "Multiple Sampling via a Time-indexed 
Method to Achieve Wide Dynamic Ranges" of David Yang et al . , 
describes a method for facilitating image multiple sampling 
using a time -indexed approach. The aforementioned patent 

3 0 and patent applications are incorporated herein by reference 

in their entireties. 

[0006] Figure 1 duplicates Figure 3 of the aforementioned 
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'786 patent application and shows a functional block diagram 
of an image sensor 300. The operation of image sensor 300 
using multiple sampling is described in detail in the '786 
patent application. Image sensor 300 includes a DPS sensor 
5 array 302 which has an N by M array of pixel elements. 
Sensor array 302 is similar to the digital pixel sensor 
described in the M25 patent and incorporates pixel level 
analog- to-digital conversion. A sense amplifier and latch 
circuit 304 is coupled to sensor array 302 to facilitate the 

10 readout of digital signals from sensor array 3 02. The 

digital signals (also referred to as digital pixel data) are 
stored in digital pixel data memory 310. To support 
multiple sampling, image sensor 3 00 also includes a 
threshold memory 306 and a time index memory 3 08 coupled to 

15 sensor array 302. Threshold memory 306 stores information 
of each pixel indicating whether the light intensity value 
measured by each pixel in sensor array 3 02 has passed a 
predetermined threshold level. In this example, the 
information is stored as a one-bit threshold indicator bit. 

20 The exposure time indicating when the light intensity 

measured by each pixel has passed the threshold level is 
stored in time index memory 308. In this example, the time 
index value is a two-bit value identifying each time 
exposure. As a result of this memory configuration, each 

25 pixel element in sensor array 302 can be individually time- 
stamped by threshold memory 3 06 and time index memory 3 08 
and stored in digital pixel data memory 310. 

[0007] With the memory configuration outlined above and 

illustrated in Figure 1, image sensor 300 can implement 
3 0 multiple sampling to improve the quality of an image. In 

multiple sampling, each pixel element is exposed to an image 
at two or more different exposure times in order to 
compensate for bright and dark portions of the image. 
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Additionally, the information regarding the exposure time 
associated with each pixel and the integrated intensity for 
that pixel is stored in time index memory 308 and digital 
memory 310 for use in computing the simulated pixel 
5 intensity when needed. 

[0008] Sensor array 302 is an N by M array of pixels 
where each pixel outputs a digitized pixel voltage signal 
having k bits. Thus, the size of threshold memory 306 is N 
by M bits and the size of time index memory 308 is N by M by 

10 m bits where m is the number of bits representing the time 
index values. For example, when the resolution of sensor 
array 3 02 is 1024 by 1024 pixels, each pixel outputting 10 
bits each (i.e., N=M=1024 and k=10) , the size of threshold 
memory 3 06 is 1 megabits, the size of time index memory 3 08 

15 with a 2 -bit time index value is 2 megabits, and digital 
pixel data memory 310 is at least 10 megabits (or 
1024x1024x10 bits) for storing one frame of image data. 

[0009] To implement multiple sampling in an image sensor, 
memory space must be provided to store image information 

2 0 such as the threshold indicator bit and the time index 

value. When image sensor 300 in the example above is 
implemented in an integrated circuit, the size of the on- 
chip memory must be at least 13 megabits. If the resolution 
of the sensor array (i.e. the number of pixel elements) 
25 increases, the size of the on-chip memory will increase 

correspondingly. Integrating a large on-chip memory in an 
image sensor not only increases manufacturing cost but also 
adversely impacts yield. Therefore, it is desirable to 
minimize the size of the on-chip memory while supporting 

3 0 multiple sampling operations in a digital image sensor. 
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SUMMARY OF THE INVENTION 

[0010] According to an embodiment of the present 
invention, an image sensor includes a sensor array, a 
companding circuit and a data memory. The sensor array 
5 includes a two-dimensional array of pixel elements and 

outputs digital signals as k-bit pixel data representing an 
image of a scene. The companding circuit is operable to 
compand the k-bit pixel data into h bits, h being less than 
k. The data memory is in communication with the sensor 
10 array for storing the h-bit pixel data for each of the pixel 
elements. In one embodiment, the circuit or method of the 
companding operation applies a transfer function which is 
linear for low intensity values and logarithm for high 
intensity values. 

15 [0011] The present invention is better understood upon 

consideration of the detailed description below and the 
accompanying drawings . 

BRIEF DESCRIPTION OF THE DRAWINGS 

[0012] Figure 1 is a functional block diagram of an image 
20 sensor as described in U.S. patent application 09/567,786. 

[0013] Figure 2 is a block diagram of a digital image 
sensor as described in U.S. Patent No. 5,4 61,42 5 of Fowler 
et al. 

[0014] Figure 3 illustrates four rows of exemplary memory 
25 cells which are used to store the 13 -bit image information 
in the memory configuration of Figure 1. 

[0015] Figure 4 illustrates the pixel intensity values 
vs. exposure time for four representative pixels A, B, C, 
and D detected by a DPS array. 



3 0 [0 016] Figure 5 is a functional block diagram of an image 
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sensor according to one embodiment of the present invention. 

[0017] Figure 6 illustrates four rows of exemplary memory 
cells in a data memory used to store the image information 
according to one embodiment of the present invention. 

5 [0018] Figure 7 illustrates four rows of exemplary memory 
cells for storing image information according to another 
embodiment of the present invention. 

[0019] Figure 8 shows a transfer function for converting 
a 10 -bit value to a 9-bit value. 

10 [0020] Figure 9 shows a transfer function for converting 
a 9-bit value to an 8-bit value. 

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 
[0021] In accordance with the present invention, a method 
and apparatus for storing image information in a digital 

15 pixel sensor is provided for reducing the size of the memory 
needed to facilitate multiple sampling. The image 
information storage method of the present invention permits 
the size of the on-chip memory in a digital image sensor to 
be reduced while preserving the image resolution. By 

20 reducing the size of the on-chip memory, the present 

invention provides the benefits of reducing manufacturing 
cost and improving production yield. 

[0022] In the present description, a digital pixel sensor 
(DPS) array or a sensor array refers to a digital image 

25 sensor having an array of photodetectors where each 

photodetector produces a digital output signal . In one 
embodiment of the present invention, the DPS array 
implements the digital pixel sensor architecture illustrated 
in Figure 2 and described in the aforementioned M25 patent. 

30 The DPS array of the M25 patent utilizes pixel level 
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analog-to-digital conversion to provide a digital output 
signal at each pixel. The pixels of a DPS array are 
sometimes referred to as a "sensor pixel" or a "sensor 
element" or a "digital pixel," which terms are used to 
indicate that each of the photodetectors of a DPS array 
includes an analog- to-digital conversion (ADC) circuit, and 
is distinguishable from a conventional photodetector which 
includes a photodetector and produces an analog signal. The 
digital output signals of a DPS array have advantages over 
the conventional analog signals in that the digital signals 
can be read out at a much higher speed. Of course, other 
schemes for implementing a pixel level A/D conversion in an 
area image sensor may also be used in the image sensor of 
the present invention. 

[0023] In the digital pixel sensor architecture shown in 
Figure 2, a dedicated ADC scheme is used. That is, each of 
pixel element 15 in sensor array 12 includes a ADC circuit . 
The image sensor of the present invention can employ other 
DPS architectures, including a shared ADC scheme. In the 
shared ADC scheme, instead of providing a dedicated ADC 
circuit to each photodetector in a sensor array, an ADC 
circuit is shared among a group of neighboring 
photodetectors. For example, in one embodiment, four 
neighboring photodetectors may share one ADC circuit 
situated in the center of the four photodetectors. The ADC 
circuit performs A/D conversion of the output voltage signal 
from each photodetectors by multiplexing between the four 
photodetectors. The shared ADC architecture retains all the 
benefits of a pixel level analog- to-digital conversion while 
providing the advantages of using a much smaller circuit 
area, thus reducing manufacturing cost and improving yield. 

[0024] In one embodiment of the present invention, the 



M- 11562 US 
740509 vl 

ADC circuit of each digital pixel or each group of digital 
pixel is implemented using the Multi -Channel Bit Serial 
(MCBS) analog-to-digital conversion technique described in 
the aforementioned 11 657. The MCBS ADC technique of the '657 
5 patent can significantly improve the overall system 

performance while minimizing the size of the ADC circuit. 
Furthermore, as described in the x 657 patent, an MCBS ADC 
has many advantages applicable to image acquisition and more 
importantly, facilitates high-speed readout. 

10 [0025] As described above, Figure 1 illustrates the 

memory configuration shown in the '786 patent application 
for facilitating multiple sampling in image sensor 300. 
Image sensor 3 00 includes threshold memory 3 06, time index 
memory 308 and digital pixel data memory 310. In image 

15 sensor 300, separate memory blocks or separate locations in 
a memory array are provided to store the threshold indicator 
information, the time index values and the digital pixel 
data. In the example given above where DPS array 3 02 
outputs pixel data in 10 bits, threshold data and time index 

20 values are stored in 1 bit and 2 bits, respectively, 13 bits 
are used to store all of the image information for each 
pixel element in DPS sensor array 302. Therefore, in the 
configuration of image sensor 3 02, a memory size of N by M 
by 13 is required to store one frame of image data and to 

25 support multiple sampling operations. For example, when DPS 
array 3 02 has 1024 by 1024 pixel elements, the on-chip 
memory is at least 13 megabits . The image storage method of 
the present invention employs innovative image information 
storage schemes to reduce the amount of memory needed to 

3 0 store all of the image information for facilitating multiple 
sampling operations in a DPS sensor array. In the following 
description, the digital pixel data, the threshold indicator 
information and the time index values generated in an image 
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sensor during the multiple sampling operations are 
collectively referred to as "image information." 

[0026] Figure 3 illustrates four rows of exemplary memory 
cells which are used to store the 13 -bit image information 
5 in the memory configuration of Figure 1. Figure 3 is 

included to illustrate the multiple sampling operation of 
image sensor 3 00 of Figure 1 and to provide a contrast to 
the innovative image information storage method of the 
present invention. Referring to Figure 3, each row of 

10 memory cells contains 13 bits for storing image information 
including the threshold indicator bit (memory cell 3 55) , the 
time index value (2-bit memory cells 357) and the digital 
pixel data (10-bit memory cells 359) . To facilitate the 
description of the present invention, a row of memory cells 

15 containing 13 bits is explicitly allocated for each pixel. 
However, it is known to those skilled in the art that other 
memory allocation schemes are possible, such as storing 
different number of bits in each row of memory cells. The 
multiple sampling operation of image sensor 3 00 is described 

2 0 in conjunction with Figure 4 which illustrates the pixel 

intensity values vs. exposure time for four representative 
pixels A, B , C, and D detected by a DPS array such as array 
302 . 

[0027] When multiple sampling is used, pixel values are 
25 first read out at an exposure time IT and a multiple 
sampling logic circuit in image sensor 3 00 performs a 
threshold comparison operation on the pixel values. The 
threshold comparison operation can be implemented in a 
variety of ways. In the present example, a 50% threshold 

3 0 level is used. Thus, at time IT, the multiple sampling 

logic circuit of image sensor 300 compares the pixel values 
readout from each pixel element of DPS array 3 02 and 
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determines which of the pixel intensity values exceeds the 
50% threshold level. For example, in Figure 4, pixel A has 
an intensity value exceeding the 50% threshold level while 
pixels B to D have intensity values below the threshold 
5 level . Pixel data for pixels A to D are recorded in data 
memory 310. Row 324 of Figure 3 illustrates the image 
information recorded for pixel A. The threshold indicator 
bit (cell 355) of row 324 is set to a value of "0" in this 
case to indicate that pixel A has reached the threshold 

10 level. The threshold indicator bit will be used to prevent 
further updating of pixel data for pixel A in digital pixel 
data memory 310. The time index value "00" associated with 
the exposure time IT is stored in cells 357 of row 324. The 
pixel data value and the time index value for pixel A will 

15 be used by image sensor 3 00 to derive the resultant 

intensity value of pixel A. The pixel data for pixels B to 
D are also stored (not shown) and the threshold indicator 
bit (memory cell 355) for each of these pixels are set to a 
value of "1" to indicate that the pixel data has not reached 

2 0 the threshold level yet and therefore, can be updated in 
subsequent exposure times. 

[0028] In Figure 4, the multiple sampling process 
continues with sensor readout being taken after exposure 
times of 2T and 4T where the time 4T is the last exposure 

25 time. Each time the pixel intensity value of a pixel 

exceeds the 50% threshold level, the threshold indicator bit 
is set to "0" and the associated time index for that pixel 
is stored in the time index memory 308. The measured 
digital pixel values are stored in pixel data memory 310. 

30 For instance, row 326 stores the image information for pixel 
B which pixel intensity value exceeds the threshold level at 
an exposure time of 2T . A time index value of "01" is 
stored in cells 357 of row 326 to indicate the exposure time 
-10- 
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of 2T. Row 32 8 stores the image information for pixel C 
which pixel intensity value exceeds the threshold level at 
the last exposure time of 4T. A time index value of XN 10" is 
stored in row 328 to indicate the exposure time of 4T. 
5 Finally, the pixel intensity value of pixel D has not 

reached the threshold level by the last exposure time 4T and 
so the threshold indicator bit for pixel D is not altered. 
The time index value of "10" is stored and the pixel data is 
recorded in row 330. In this manner, 13 bits of image 
10 information is used to facilitate the multiple sampling 
operations in image sensor 3 00. 

[0029] Turning now to the image information storage 
method for facilitating multiple sampling of the present 
invention, Figure 5 is a functional block diagram of an 

15 image sensor 400 according to one embodiment of the present 
invention. In the present description, like objects which 
appear in more than one figure are provided with like 
reference numerals to simplify the discussion. Image sensor 
400 includes a DPS sensor array 3 02 which operates in the 

2 0 same manner as previously described and provides digital 
pixel data as output signals. Image sensor 400 further 
includes a data memory 410 which integrates the storage of 
the threshold indicator information, the time index values, 
and the pixel data. The image information storage method of 

2 5 the present invention minimizes the amount of memory needed 

in memory 410 to facilitate multiple sampling. Of course, 
image sensor 400 may include other circuitry such as decoder 
312 and sense amplifier and latch circuit 3 04 which operate 
in the same manner as image sensor 3 00. 

3 0 [0030] Figure 6 illustrates four rows of exemplary memory 

cells in data memory 410 used to store the image information 
according to one embodiment of the present invention. In 
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the present embodiment, only 11 bits are needed to capture 
all of the image information for each pixel element of DPS 
array 3 02 in image sensor 4 00, as opposed to 13 bits 
required for image sensor 3 00. The present embodiment is 
5 thus referred to as the "11-bit image information 

embodiment." Thus, data memory 410 can have a smaller 
memory size than the size of the total memory in image 
sensor 300. Specifically, a total of NxMx2 bits of memory 
cells is saved by using the image information storage method 
10 according to the present embodiment of the present 

invention. Thus, image sensor 400 can be made with a 
smaller device size and can realize reduced cost and 
increased production yield. 

[0031] In accordance with the present invention, image 
15 sensor 4 00 encodes the threshold indicator information in 
the time index values and alternates between storing the 
entire time index value or a portion of the time index value 
only to make room for storing an additional bit of pixel 
data in data memory 410. In this manner, image sensor 400 
20 makes efficient use of the memory space in data memory 410. 
In operation, in the last exposure time, image sensor 4 00 
stores the time index value using only a portion of the 
assigned memory location and image sensor 400 uses the 
remaining memory location to store additional pixel data 
25 information. Encoding the threshold indicator information 
in the time index values eliminates the need to use a 
separate threshold indicator bit. Therefore, in accordance 
with the present embodiment, only 11 bits total are needed 
to store all the image information needed. 

30 [0032] Referring to the 11-bit image information 

embodiment in Figure 6, data memory 410 includes memory 
cells memory cells 460 and 462 in each of rows 424 to 430 
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designated to store a 2 -bit time index value. Memory cells 
464 are designated to store 9 bits of pixel data. The 2 -bit 
time index value stored in cells 460 and 462 also includes 
encoded threshold indicator information. In the present 
5 embodiment, the threshold indicator information is encoded 
in the first bit of the two-bit time index value, that is, 
cell 460. Of course, in other embodiments where the time 
index value is stored in more than 2 bits, the threshold 
indicator information may be encoded in one or more bits of 

10 the time index value. According to the present embodiment, 
in the "all-other exposure cases," data memory 410 stores 
the time index values in two bits and the pixel data in 9 
bits. The term "all-other exposure cases" is used to refer 
to situations where the pixel intensity value of a pixel 

15 element exceeds the predetermined threshold level at any one 
of the exposure times before the last exposure time . In 
those cases, it is necessary to retain the time index values 
as well as the pixel data so that the pixel data for the 
pixel element can be normalized later on to provide a 

2 0 resultant pixel data value. 

[0033] On the other hand, in the "last exposure case" 
where the pixel intensity value of the pixel element has not 
exceeded the threshold level before the last exposure time, 
the pixel data is captured at the last exposure time and is 
25 stored as a 10-bit pixel data in data memory 410. According 
to the present embodiment, in the last exposure case, only 
one bit of the time index value (cell 460) is stored and 
the remaining bit (cell 462) in the time index value is 
provided to store an additional bit of pixel data. Thus, in 

3 0 the last exposure case, both the time index value and the 

threshold indicator information are encoded in memory cell 
460 and memory cell 462, the second bit of the time index 
value, is combined with the 9 -bit memory cells 464 to 
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provide memory cells for storing the 10 -bit pixel data. In 
this manner, the 11 -bit image information embodiment 
provides the same number of data bits for storing pixel data 
as in the prior method shown in Figure 3. Thus, the 10-bit 
pixel data can be perfectly represented. In Figure 6, the 
11-bit image information is shown as being arranged in a 
contiguous fashion. Of course, this is illustrative only 
and one of ordinary skill in the art would understand that 
memory cells 460, 462 and 464 need not be allocated in a 
contiguous manner and in actual implements, can be arranged 
in any fashion which facilitates memory access for reading 
and writing. In fact, it is known to those skilled in the 
art that various other memory allocation schemes can be used 
for storing memory cells 460, 462 and 464 in data memory 
410. For example, data memory 410 can store different 
number of data bits in each row of memory cells in the 
memory array. In this manner, data memory 410 provides 
flexibility in storing pixel image information and provides 
efficient use of the memory cells in data memory 410 . 

[0034] The operation of the image information storage 
method of the present invention will now be described in 
conjunction with Figure 4. When multiple sampling is used 
in image sensor 4 00, pixel values are read out at exposure 
times of IT, 2T and 4T, where exposure time 4T is the last 
exposure time. Exposure times IT and 2T are represented by 
time index values "00" and "01" respectively. The exposure 
time 4T is represented by the time index value "1" in the 
first bit of the 2 -bit time index value. The threshold 
level is set at 50% in the present example. For pixel data 
represented in 10 bits, the 50% threshold level represents a 
pixel intensity value of 512. Row 424 stores the image 
information for pixel A which intensity level exceeded the 
threshold level at the exposure time IT. Assume that pixel 
-14- 
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A has an intensity value of 780 at exposure time Tl, image 
sensor 400 determines that pixel A has exceeded the 50% 
threshold level and proceeds to store the image information 
for pixel A. The time index value xx 00" representing 
5 exposure time Tl is stored in cells 460 and 462 of row 424. 
The value of " 0" in cell 4 60 (the threshold indicator) 
functions to indicate that pixel A has exceeded the 
threshold level and prevents further updating of the pixel 
data. The lower 9 bits of the 10 bit pixel data for pixel A 
10 is stored in memory cells 464 of row 424. That is, a value 
of 268 in 9-bit representation is stored for pixel A. Thus, 
according to the present embodiment, only 11 bits are needed 
to store all of the image information for a pixel element in 
image sensor 400 without compromising the image quality. 

15 [0035] In accordance with the present embodiment, even 
though only the lower 9 bits of the 10-bit pixel data are 
stored, the pixel value is preserved. This is because the 
10 th bit, or the most significant bit (MSB) , of the pixel 
intensity value is necessarily a "1" when the pixel 

20 intensity value exceeds the 50% threshold level. Thus, the 
MSB of the pixel data need not be stored but instead can be 
reintroduced later based on the time index values. In the 
present example, a pixel value of 780 is represented in 
binary as "1100001100" while the lower 9 bits provides a 

25 value of 268 and the 10 th bit provides a value of 512 which 
is the threshold value. In the present embodiment, a value 
of 268 in 9 bits is stored in memory cells 464. 

[0036] When the resultant pixel value for pixel A is to 
be computed later by image sensor 400 (or other image 
3 0 processing device coupled to image sensor 4 00) , image sensor 
400 retrieves the pixel data stored in row 424 of memory 410 
and recognizes that pixel A has exceeded the threshold level 

-15- 



M-115S2 US 
740609 vl 

at time Tl based on the time index value "00" stored in 
cells 460 and 462. Consequently, image sensor 400 will 
automatically add the threshold value 512 to the pixel value 
stored in memory cells 464 of row 424 before computing the 
5 resultant pixel value for pixel A. Thus, the pixel value of 
78 0 recorded at exposure time T associated with pixel A is 
preserved entirely. The resultant pixel value for pixel A, 
that is, the pixel value for pixel A at the last exposure 
time, is computed as 4 times 780. 

10 [0037] Turning now to pixel B, row 426 stores the image 

information for pixel B which exceeds the threshold level at 
exposure time 2T. Thus, a time index value of "01" is 
stored in memory cells 460 and 462. The lower 9 bits of the 
pixel data of pixel B are stored in memory cells 464 of row 

15 426. The storage and retrieval of pixel data value for 

pixel B is analogous to pixel A. The measured pixel value 
less 512 is stored as a 9-bit number. Then, when the 
simulated pixel data for pixel B is to be computed, a value 
of 512 is added back to the 9-bit number stored in cells 464 

2 0 of row 42 6 to arrive at the actual measured pixel value for 

pixel B. 

[0038] Turning now to pixels C and D where in exposure 

times IT and 2T, the pixel data have not exceeded the 
threshold level, the pixel data in 10 bits are written in 
25 rows 428 and 430 (not shown) and a value of "1" is written 
in the threshold indicator (cell 460) . In this manner, the 
value of "1" indicates that the pixel data for these pixels 
have not exceeded the threshold level and that the pixel 
data for pixels C and D can be further updated in subsequent 

3 0 exposure times. In Figure 4, pixels C and D have not 

exceeded the threshold level before the last exposure time 
4T. Thus, the pixel data values of pixels C and D at 
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exposure time 4T are stored as 10 -bit pixel data. Memory 
cell 460 of each of rows 428 and 430 maintains the value of 
"1", indicating that the threshold level has not been 
previously exceeded. Of course, memory cell 460 can also be 
rewritten with the value of " 1" . The writing of memory cell 
460 in each of the exposure time is optional. In other 
embodiments, memory cell 46 0 can be written at each exposure 
times regardless of the pixel data value or memory cell 460 
can be written at the first exposure time only and updated 
only when the pixel data exceeds the threshold level and the 
time index value is stored. Memory cell 462 and 9 -bit 
memory cells 464 are used to store the 10 -bit pixel data. 
In the last exposure case, all 10 bits of the pixel data 
needs to be stored because the pixel data can have a value 
larger than or less than 512. For instance, pixel C has a 
pixel value exceeding the 50% threshold value and pixel D 
has a pixel value less than the 50% threshold value. In 
either case, the 10 bits pixel data for pixels C and D are 
stored in their respective rows 428 and 430. When the pixel 
data are retrieved later on, the value of "1" in the 
threshold indicator (cell 460) indicates that the pixel data 
in rows 42 8 and 43 0 have not exceeded the threshold level 
before the last exposure time and that the pixel data stored 
are 10 bits pixel data. 

[0039] In the embodiment shown in Figure 6, the image 
information storage method of the present invention achieves 
a 2 -bit reduction in memory size for each pixel element. 
Thus, in DPS array 3 02 having N by M pixel elements, a 
reduction in overall memory size of NxMx2 bits is achieved. 
When DPS array 3 02 is a 1024 pixels by 1024 pixels array, a 
memory size reduction of 2 megabits can be realized. Image 
sensor 400 thus constructed has a smaller device size, 
resulting in lower manufacturing cost and improved yield. 
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[0040] In the present embodiment, the pixel data 
retrieval and computation process can be implemented either 
in hardware or in software. Thus, in one embodiment, logic 
circuits are included in image sensor 400 for computing the 
5 pixel data upon retrieval of the pixel data from data memory 
410. In another embodiment, the time index values and the 
pixel data can be read out of image sensor 40 0 into an image 
processing device where the pixel data are computed using 
software. Normalization of the pixel data can be carried out 

10 after the pixel data value is computed. Thus, in a software 
implementation, the pixel data in data memory 410 is read 
into a variable . This variable is then applied to a lookup 
table which matches the value of the variable to a 12 -bit 
integer representing the normalized pixel data value. This 

15 12 -bit integer can then be used in the remainder of the 
imaging pipeline. 

[0041] According to another embodiment of the present 
invention, the image information storage method provides 
further reduction of the number of bits used to store image 

2 0 information. The further reduction is accomplished through 

companding of the pixel data. Although in the present 
description, companding of pixel data is described in 
conjunction with the use of multiple sampling in a digital 
image sensor, companding of pixel data can be used even when 
25 the image sensor is not using multiple sampling. As will be 
described in more detail below, companding can be used to 
compress the pixel data so that fewer number of data bits is 
required to represent the pixel data. Companding can be 
applied in an image sensor for reducing the memory storage 

3 0 space required whether or not multiple sample is applied. 

[0042] Figure 7 illustrates four rows of exemplary memory 
cells for storing image information according to another 
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embodiment of the present invention. In the embodiment 
shown in Figure 7, a total number of 10 bits is used to 
store the image information, as opposed to the 11 bits of 
the previous embodiment. The present embodiment is referred 
5 to as the "10-bit image information embodiment." The 1-bit 
reduction from the embodiment shown in Figure 6 is achieved 
through companding of the pixel data generated by each pixel 
element so that a fewer number of bits of pixel data is 
stored. In the present embodiment, companding is performed 

10 to convert the 9 bit pixel data generated for the all -other 
exposure cases (pixels A and B of Figure 4) into an 8 -bit 
representation. That is, in the cases where the pixel value 
for a pixel exceeds the threshold level prior to the last 
exposure, the lower 9 bits of the pixel data value are 

15 compressed into 8 bits and stored as an 8 -bit value. In the 
last exposure case (pixels C and D of Figure 4) , companding 
is performed to convert the 10 bit pixel data into a 9 -bit 
representation. Thus, in cases where the pixel value of a 
pixel element has not exceeded the threshold level prior to 

20 the last exposure, the 10 -bit pixel data captured at the 

last exposure time is compressed into 9 bits and stored as a 
9-bit value. 

[0043] The 10 -bit image information embodiment shown in 

Figure 7 realizes a further 1-bit reduction in memory size 

25 for each pixel element than the 11 -bit embodiment and 

provides a 3 -bit reduction in memory size for each pixel 
element when compared to the previous method described in 
Figure 3 . When the 10-bit image information embodiment is 
applied in image sensor 400, data memory 410 can have a size 

3 0 of N by M by 10 bits only. When DPS array has 1024 pixels 

by 1024 pixels, data memory 410 only needs to be 10 megabits 
to implement the 10 -bit embodiment for supporting multiple 
sampling operations. 
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[0044] Companding, derived from compressing and 
expanding, is a well-known compression technique which uses 
a non- linear transfer function for the treatment of voice 
samples. For instance, a function for applying companding 
5 provides finer spacing at low volume and wider spacing at 
the loud end. In accordance with the present invention, 
companding is applied to image data by exploiting the 
characteristics of human visual perception. Human visual 
perception is much more acute at low light conditions than 

10 at bright light conditions. Specifically, human eyes can 
only perceive changes in an image when the changes in the 
intensity level of the image exceed a certain percentage of 
the intensity level of the image (the percentage is referred 
to here as the "perceptible threshold"). Thus, under bright 

15 light conditions, the human eyes can only perceive large 
variations in intensity values, while under low light 
conditions, the human eyes can perceive smaller variations 
in intensity values. Accordingly, in the present invention, 
companding is applied to compress large pixel intensity 

2 0 values representing bright light conditions while preserving 

the small pixel intensity values representing low light 
conditions . 

[0045] When the 10 -bit image information embodiment is 
used in image sensor 400, the image quality at bright light 
25 portions of the image may be compromised to a small extent, 
but there is little or no impact to the image quality at the 
medium to low light portions of the image. Since the human 
visual perception is not particularly acute in bright light 
areas anyway, the viewer may not be able to perceive any 

3 0 degradation in the overall image quality. Furthermore, the 

benefits obtained from reducing the size of the data memory 
to achieve lower cost and improved yield outweighs the de 
minimis image degradation which can be practically 
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perceived. 

[0046] Figures 8 and 9 depict two exemplary transfer 
functions which can be used in the companding operations of 
the present invention. Figure 8 shows a transfer function 
5 for converting a 10 -bit value to a 9 -bit value. The 

transfer function (depicted by curve 602) is linear at low 
intensity values (such as below intensity value 300) . The 
slope of curve 6 02 tapers off at high intensity values so 
that 10 -bit values from 0 to 1023 are mapped to 9 -bit values 

10 from 0 to 511. In the present embodiment, the increment 

value of curve 602 is an integer and has an increment value 
of 1 at low intensity values and an increment value of less 
than 0.7% at high intensity values. Of course, other 
transfer functions can also be used in the companding 

15 operations of the present invention. In other embodiments, 
the transfer function can be defined as follows . A 
percentage increment value for the transfer function is 
selected which is below the perceptible threshold of human 
visual capability. The transfer function is derived by 

2 0 stepping through the 10 -bit values using the percentage 

increment value and mapping the 10 -bit intensity values to 
9-bit intensity values. For low intensity values, the 
increment value can be rounded up to an integer to preclude 
the use of unnecessarily fine increment values. For 

2 5 example, when a percentage increment value of 1% is used, at 
low intensity values, the 10-bit values will increment by 1 
while at high intensity values, the 10 -bit values will 
increment by 1%. Thus, for an intensity value of 10, the 
step size is 1. For an intensity value of 500, the step 

30 size is 5. A transfer function such as curve 602 of Figure 
8 can be thus formed. 

[0047] Figure 9 illustrates a transfer function (curve 
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608) which can be used for implementing the 10-bit to 8-bit 
companding operation according to one embodiment of the 
present invention. According to the present invention, the 
10 -bit to 8 -bit conversion is performed only for the all- 
5 other exposure case where the pixel intensity values have 
exceeded the threshold level which is 512 in the present 
example. Thus, in the present embodiment, only pixel 
intensity values between 512 and 1023 are mapped to the 8- 
bit values from 0 to 255. Note that pixel intensity values 

10 between 512 and 1023 can be represented in 9 bits as the MSB 
of these values is always a "1." Thus, curve 608 can also 
be treated as a 9 -bit to 8 -bit companding transfer function. 
Referring to Figure 9, curve 6 08 maps pixel values from 4 54 
to 1023 to 8-bit values from 0 to 256. In the present 

15 embodiment, the 10 -bit to 8 -bit conversion uses the top 55% 
of the 10 -bit values instead of the top 50% (from 512 to 
1023) . Of course, in other embodiments, the 10 -bit to 8 -bit 
transfer function can be provided for the top 50% of the 10- 
bit values only. Also, in other embodiments, the transfer 

20 function can be provided to map 9-bits values from 0 to 511 
to the 8-bit values from 0 to 255. In that case, the MSB of 
the pixel data is discarded and the lower 9 bits of the 
pixel data are used in the companding operation. 

[0048] The transfer functions for the companding 
25 operations can be implemented using a look-up table. 

According to one embodiment of the present where the MCBS 
analog- to-digital conversion technique is used in the pixel - 
level ADC circuit, the look-up table is used in the ADC 
circuit for programming the ramp signal to the comparator. 
30 In yet another embodiment of the present invention, the 10- 
bit image information embodiment can be selectively applied 
by programming the ramp signal with either an entirely 
linear function (no companding) or with a transfer function 
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as shown in Figures 8-9. 

[0049] When the 10-bit image information embodiment is 
used to store image information, the pixel data can be 
retrieved by applying the companding transfer function in 
5 reverse. For example, when the pixel value for pixel A 

stored in row 524 is to be retrieved, the time index value 
of "00" in cells 460 and 462 indicates that the pixel value 
has exceeded the threshold level at exposure time T. The 8- 
bit pixel value in memory cells 564 is mapped using the 

10 transfer function (curve 608) in Figure 9 in the reverse 

manner to a 10 -bit pixel value. In another embodiment when 
the transfer function maps the 9-bit pixel values to 8-bit 
values and the threshold level is at 50%, when the transfer 
function is applied in reverse, the threshold value 512 is 

15 added back onto the 9 -bit pixel value to reflect the actual 
pixel intensity value of pixel A in 10 bits. In the last 
exposure case when pixel values for pixels C and D are to be 
retrieved, the value of "1" in the threshold indicator cell 
460 signifies that the pixel values have not exceeded the 

2 0 threshold level prior to the last exposure. The 9 -bit pixel 

values in rows 52 8 and 530 are mapped back to the 10 -bit 
value using the transfer function in Figure 8 {curve 602) . 
The pixel values can then be normalized and processed as 
desired. In cases where the compression in the companding 
25 operation causes one 9-bit value or one 8-bit value to map 
to two or more 10-bit values, one of the 10-bit values can 
be chosen by ensuring that the 10 -bit pixel values are 
spaced apart appropriately. 

[0050] In the above described embodiments, a two-bit time 

3 0 index value is used to support three exposure times. When 

the time index value is two bits, the multiple sampling 
operation according to the present invention can include 
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three exposure times {i.e., T, 2T and 4T) or less. Of 
course, in other embodiments of the present invention, the 
time index value can include any number of bits to support 
any desired number of exposure times. Specifically, the 
5 relationship between the number of bits, m, of the time 
index value and the number of exposure times which can be 
supported by the method of the present invention is given as 
follows : 

No. of exposure times < 2 m -l. 

10 Thus, when the time index value has 3 bits (m=3) , the number 
of exposure times which can be supported is 7 or less. When 
the time index value has 4 bits (m=4) , the number of 
exposure times which can be supported is 15 or less. As 
mentioned above, when the time index value is represented in 

15 m bits, the threshold indicator information can be encoded 
using one or more bits of the time index value. 

[0051] The above detailed descriptions are provided to 
illustrate specific embodiments of the present invention and 
are not intended to be limiting. Numerous modifications and 

2 0 variations within the scope of the present invention are 

possible. For example, while in the above described 
embodiments, companding is performed to reduce the number of 
bits of the pixel data by 1 bit, it is, of course, possible 
to use companding to reduce the number of bits in the pixel 
25 data by more than 1 bit. Of course, companding more than 1 
bit may result in a loss of image information, particularly 
at bright light conditions. The amount of companding which 
can be used is a function of the application the image data 
is being applied to. For certain applications, the loss of 

3 0 image information may not be important and may not impact 

image quality. Thus, companding of more than 1 bit can be 

used to further reduce the size of the data memory needed in 
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the image sensor. Furthermore, while in the above 
description, the multiple sampling operations used a 50% 
threshold level, one of ordinary skill in the art would 
appreciate that other values of threshold level can be used, 
5 such as 22% or 65%. The present invention is defined by the 
appended claims . 
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